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ABSTRACT 

Cr iterion- referenced testing is defined and sere of 
its background is discussed. A distinction is made between 
c r iter ion- ref erencod scores, norm-referenced scores, cut-off scores, 
criterion scores, criterion variables, and content- st anda rd scores. 
The relationship between norm-referenced information and 
criterion-referenced information is considered. The need for 
vigorous, emp ir leal ly-ba see construct validation studies of 
criterion-referenced tests ’ s pointed on*-. The use of 

or iter ion- ref erenced testino in instruction is considered in terms of 
absolute interpretations and mastery learning. It is seen that 
/hither criterion-ref ere uoed testing and/or norm- r ef erenced testina 
is needed to make instructional decisions defends upon the 
instructional context within which one opeiates. It is concluded 
that, for purposes of instruction and instructional decision-making, 
there is a need for the integration of measurement Vnowledce with 
knowledge about instructional psychology. {Author/TAl 
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Abstract 



Criterion-Referenced Testing in the 
Context of Instruction* 

Anthony J. Nitko 
University of Pittsburgh 

Criterion-referenced testing is defined and some of Its background 
is discussed. A distinction is made between criterion-referenced scores, 
norm-referenced scores, cut-of* scores, criterion scores, criterion vari- 
ables, and content- standard scorec . The relationship between norm-referenced 
information and criterion-referenced information is considered, The need 
for vigorous, empirically-based construct validation studies of criterion- 
referenced teats is pointed out. The use of criterion-referenced testing 
in instruction is considered in terms of absolute interpretations and mastery 
learning. It is seen that whether criterion-referenced testing snd/or norm- 
referenced testing is needed to make instructional decisions depends upon 
the ini -ruct ional context within which one operates. It is concluded that, 
for purposes of instruction and instructional decision-making, there is 
a need for the integration of measurement knowledge with knowledge about 
Instructional psychology. 



The preparation of this paper was supported by the Learning Research 
fjid Development Center supported au a research and development center by 
funds from the United States Office of Education, Department of Health, 
Education, and Welfare. 



Criterion-Referenced Testing in the 
Context of Tnstruction* 

When we taLk about criterion-referenced testing, we need to 
distinguish it from some traditional usages of the word criterion with 
which it tends to be confused. The term criterion hat? been used many 
times in psychometrics to refer to a second variable which we are inter- 
ested in predicting. For example, an aptitude test is sometimes said to 
predict a criterion such as end of course grades or scores cn an achievement 
test* Sometimes the validity of a test is described in terms of its cor- 
relation with some criterion (or criteria). 

A second common usage of criterion has been that of criterion scores* 
The criterion scoxe functions much the same as a cut-off score for some 
decision* In this context, expressions such as "working to criterion 
level" have been employed. For example, a statement like: "this student 

answered 50 per cent of the teat questions correctly, but has not reached 
the criterion level of performance which is answering 85 per cent of the 
questions correctly*" 

Neither of these two usages of the term criterion is quite what Is 
meant by criterion-referenced testing* It is useful, therefore, to review 
some of the background for criterion-referenced testing in order to more 
clearly describe It. 

^Grateful acknowledgement is made to Robert Glaser and Richard L, 
Ferguson for theiv* helpful comments on the draft manuscript* 
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Criterion-Referenced Testin g 

Although It may be true that criterion-referenced testa were 
used earlier, the term can probably be attributed to Robert Glaser. It 
was first mentioned in connection with proficiency measurement in training 
(Glaser and Klaus, 1962) and later was applied to the measurement of educa- 
tional achievement (Glaser- 1963). The motivation for this application to 
achievement measurement stemmed from a concern about the kind of achieve- 
ment information required to make instructional decisions. Some instruc- 
tional decisions concern individuals. For example, what kind of competence 
an individual needs in order for him to be successful in the next course 
in a sequence. Other decisions center around the adequacy of the instruc- 
tional procedure itself. Tests which provided achievement information 
about an individual only in terms of how the individual compared with other 
members of the group, or which provided only sketchy information about the 
degree of competence he possessed with respect to some desired educational 
outcome, were not sufficient to make the kinds of decisions necessary 
for effective instructional design snd guidance. 

In his dlacusalon, Glaser refers to two other people who hed 
proposed similar ideas: John Flanagan (1931) and Robert Ebel (1962), 

Both the Flanagan and Ebel ideas, while similar to Glaser’s, are different 
enough to warrant discussion. 

The Flanagan reference is to his chapter on units, scores, and 
norma in Lindquist’s (1951) E ducational Measurement . Flanagan distinguished 
between five types of descriptive information that are necessary in order 
to interpret broadly educational achievement data. In that discussion he 
made a distinction between a "standard of performance" and a "norm-performance, 1 
A standard of performance on a test is defined as a dealrable model or a 
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minimum goal we would like an individual to attain. A "norra-performnoce" 

is the present average performance or attainment with respect to a specific 

group or population. For example, 

The score of an individual as obtained on a French reading 
teat might be at the tenth-grade norm. This gives little 
information about how well he reads various types of materials. 

The probable degree of comprehension of the individual i.i 
reading a typical French newspaper would provide a useful 
social standard for interpreting scores on a French reading 
test (pages 698-699). 

He cautioned that it was unwise to use automatically and uncritically 
the present average test performance as the acceptable score for 
that test. The most fundamental piece of information that an achieve- 
ment test should provide is a description of an individual^ performance 
with respect to some defined body of content that can be interpreted with- 
out reference to the scores of other individuals or to norm groups. 

Professor Ebel (1962) extended this distinction and presented two 
schemes for developing tests whose scores could be Interpreted objectively 
and meaningfully without the use of norms. Of special emphasis are the 
content categories that the test items represent. One method would 
result In a display o' selected test items along with descriptive infor- 
ma'lc . about how many of these itam9 could be answered correctly by indi- 
viduals at various total test score levels. For example, if 10 of the 50 
mathematics items from the PSAT were displayed, it would be possible to 
make a statement like: "Persons with a standard score of 500 on the 

mathematics section of the PSAT will, on the average, get 4 or 5 of these 
10 itums correct." The selected items are obtained by first sorting a 
large number of items into subject-matter content categories, such as, 
calculations with fractions, verbal problems, triangles, circles, and so 
on. Then the one item in each category that best discriminates between the 
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high and low scoring groups on the entire tost is selected to represent 
the content category. Data for assigning meaning to a score of 500 ie 
obtained by finding how many of the ten items were answered correctly, on 
the average (the mode in this case), by those persons who had standard scores 
of 500. This is repeated for each standard score level* 

A second more basic procedure for obtaining meaningful scores 
is to make the process by which the test is constructed systematic and 
explicit. This calls for a systematic sampling of test items, rather 
than a subjectively chosen collection of tanks. For, "unless the score 
Is based on a systematic sample from a defined domain of tasks, it cannot 
provide a very sound basis for inferences as to the examinees 1 performance 
on similar collections of tasks (page 16). 11 As an Illustration, tests 
ware built that required the examinee to match definitions with words. 

"The tests ware based on a spaced sample of 100 words from a 
specified dictionary. Explicit instructions were given 
fto the teit constructors] for choosing a unique but repre- 
sentative sample, and for limiting the sample to words 
appropriate for the teat* For each word the first synonym 
or defining phrase was copied from the dictionary . * * . 

These tests constitute one operational definition of the 
proportion of words in a certain dictionary for which a 
person ’knows 1 the meaning, and hence the size of his 
vocabulary in a certain sense (pages 24-25)." 

The term "content-standard scores" was used to refer to the 
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kind of scores derived from these tests. "Content" means that the score 
is based directly cm the items comprising the test. "Standard" means both 
the common scale on which the scores are reported (per cent in this case) 
and the fact that the process by which the test ie constructed, admin- 
istered, and scored is made explicit and objective. Thus, an individual's 
obtained score is referred directly to the domain of content for inter- 
pretation* This la contrasted to normative-standard scores which are 
Interpreted by referring to the performance of other Individuals. It 
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should be noted that this Is a different use? of the vord standard than 
was used by Flanagan, who used it in the sense of a minimum goal or a 
desired model. 

In a way, Glaser (1962) combined both the notion of a desired 
model and the notion of a standard domain of content* He. called for the 
specification of the type of behavio r the individual is required to demon- 
strate with respect to the content. "The standard [or criterion 1 against 
which a student’s performance ia compared ... is the behavior which 
.defines each point alonn the achievement conci.nuum (page 519) , M A 
criterion-ref e renced test, then, is one _that _ls deliberately constructed 
to give sco res that t ell what kin d s of beer vior Individuals with those 
scores can demonstrate (Glaser and Nitko, 1970). 

As an illustration, consider the problera of assessing the com- 
petency of a student in elementary school geometij . Competency in ele- 
mentary geometry can be analyzed into a number of behavior classes. A 
test can be constructed to measure these behaviora and to give scores 
that can be interpreted in terms of them* On such a test, a score of 30 
might mean that along with a number of lower level behaviors, the student 
is able to 

identify pictures of open continuous curves, lines, line 
segments, and rays; can state how these are related to each 
other; and can write symbolic name9 for specific illustrations 
of them. He can identify pictures of intersecting and non- 
intersecting lines and can name the point of intersection. 

This score would also mean that the student could noj demonstrate higher 

level behaviors such as 

identifying pictures that show angles; naming angles with 
three points; identifying the vertex of a triangle and an 
angle; identifying perpendicular lines; use a compass for 
bisection or drawing perpendiculars; and so on. 
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In like manner, a score of 20 might mean that the student could not^ 
demonstrate any of the behaviors Implied by the higher scores, but could 
demonstrate all lo^er level behaviors, up to and including behaviors such as 

naming the plane figures that comprise the faces of cubes, 

cones, pyramids, cylinders, and prisms; naming these solids; 

and identifying pictures of these solids. 

It is apparent, then, that there are four characteristics inherent 
In criterion-referenced tests: 

(1) the classes of behaviors that define different achieve- 
ment levels are specified as clearly as is possible before 
the test is constructed . 

(2) each behavior class is defined by a set of test situations 
(that is, test items or test tasks) in vhich the behaviors 
can be displayed in terms of all their important nuances. 

(3) given that the classes of behavior have been specified 
and that the test situations have been defined, a 
representative sampling plan is designed and used to 
select the test tasks that will appear on any form of 
the test. 

(4) the obtained score must be capable of expressing objec- 
tively and meaningfully the individual's performance 
characteristics in these classes of behavior. 



Norm-Referenced Scores from Criterion-Referenced Tests 

Norm-referenced testing is well known. When a test is constructed 
to yield scores that can be interpreted in such a way as to determine 
an examinee's relative location in a population or group of other examinees 
who took the same test, then ve have a norm-referenced test. Scores 
derived for norm-referenced information are reported as percentiles, 
standard scores, grade-equivalents or age-equivalents. To obtain these 
scores, the mean, standard deviation, and sometimes the form of the dis- 
tribution is pre-specified. 

It should be obvious that criterion-referenced testing can yield 
norm-referenced information. Under certain circumstances both criterion- 

O 

referenced Information and nom-reterenced information are needed to make 




a broad interpretation of an individual's test performance. Flanagan, 

Ebel, and Glaser all point this out. 

In most circumstance one or the other kind of information is 
of primary concern. The test constructor can choose to maximize either 
criterion-referenced information or norm-referenced information, but 
seldom can he maximize both. Since norm-referenced scores derive most of 
their meaning from distributions in which we can distinguish one individual 
from another, judicious selection of test items with the help of statistical 
analysis will maximize this distinction. Such statistical selection of 
items for criterion-referenced tests makes little senre, however. The 
classes or domains of tasks which define a behavior are determined, inso- 
far as is possible, before the test is constructed and then representative 
samples are drawn for inclusion on any test. To screen out some items for 
inclusion on a particular test because they possess desirable statistical 
characteristics will change the definitions of the behavioral categories 
(cf. Osb'irn, 1968). The kind of information desired when criterion- 
referenced tests are used is the behaviors an individual does or does not 
possess and whether or rot the test yields meaningful normative-standard 
scores is often of secondary importance. 

The Need for a Data Base 

When one proceeds to build a criterion-referenced test he needs 
to be Just as rigorous as when constructing a norm-referenced test. Given 
that the classes of behavior have been defined, empirical evidence is 
needed to support any contentions that the clssses of test tasks do 
indeed reflect the behavior or competence of Interest. There is a need 
for knowledge about test construction to become integrated with psychological 
knowledge and theory. 
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More often than not, a single, verbal statement of a behavior 
Implies that an individual ought to be able to perform quite a large domain 
of tasks. This is particularly true of instructional objectives, where 
generalization and transfer are of primary importance. These domains of 
tasks need to be systematically examined and, if necessary, stratified so 
that representative sampling can take place. 

Most useful instructional objectives which are employed in curriculum 
design appear to be formulated is constructs* This is true because (1) the 
behavior that is referred to i* most often stated in terms of a class of 
responses to a class of sttoM and (2) all of these statements are often 
tied together with psychological interpretations such as the need for pre- 
requisites and the relationships among the objectives in the sequence of 
instruction. Specifications of the instructional objectives which are 
needed for criterion-referenced tests tend to avoid broad trait construct 
statements such as "reading ability.” Thus, the job of building tests that 
have representative tasks defining classes of behavior becomes more difficult 
as the behaviors become more complex. It is easier to build tests to measure 
decoding skills than to measure reading comprehension. The basis for infer- 
ence about "reading ability,” for example, is observable performance on 
the specified domain of tasks into which reading ability can be analyzed, 
such as, reading certain types of passages aloud, identifying objects 
described in a text, rephrasing sentences in a certain way, carrying out 
written instructions, reacting emotionally to described events, and so on, 

Tt would seem, then, that criterion-referenced test builders need to conduct 
many of the same kinds of construct validation studies as have been recom- 
mended for psychological tests and other kinds of achievement tests (Cronbnch 
and Keehl, 1955; Cronbach, 1969). 
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Absolute Interpretation of Tent Scores 



Recently, Cionbach (1969) has called attention to the need for 
absolute interpretations of test performance. Criterion-referenced testing 
implies this also. Absolute interpretation refers to making judgments 
about a person's score in terms of what his performance on the test is and 
what that performance represents with respect to a defined domain of 
test tasks. It is contrasted to comparative or relative interpretations, 
by which Judgments about a person's score are based on the scores of other 
individuals in the population or group to which he has membership. It is 
clear that the testing movement has given little attention to absolute 
interpretations (Cronbach, 1969). 

Absolute interpretations can be extremely dangerous, however, if 
they are used inappropt lately. Tests for which the domain of items is 
vaguely defined, for which the behaviors ellicted are indeterminate, 
and for which a representative sampling plan has been unspecified, are 
poor banes upon which to interpret scores in an absolute sense. Failure 
to perfotm proper analysis before test construction often leads to assessing 
only those educational goals that are easily measured* Such abuses are 
probably common in many classroom test ini jrpretatlons — and, perhaps, 
in much of what is currently pas Bing for criterion-referenced testing! 

As Professor Ebel < 1962 j. 1970) prints out, such abuses are reminiscent 
of the criticisms of the percentage course grade and of objective testing 
early in this century. 

These abuses then point more strongly toward the need for properly 
constructed criterion-referenced tests, based on well defined and instruc- 
tional^ meaningful behaviors, in situations where absolute Interpretations 
tend to be made or where these interpretations need to be made. This means 
replacing much of the "art" of item writing with the technology of item 
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writing: behavioral and task analysis, task construction, and domain 

specification, b'uch work is certainly not easy, but neither does it seem 
Impossible. A few notable suggestions al/rg these lines have been provided 
by Gsgn/ (1969), Hively (1966; Hively, Patterson, and Page, 1968) and 
Bormuth (1970). 

Mastery 

Criterion-referenced tests have been employed most often in 
instructional situations where the notion of mastery learning is advocated. 
One issue in which criterion-referenced testing has become entangled 
is that of determining mastery. Some propose that a cut-off or "criterion 
score" needs to be established and that each student must be taught until 
ha obtains a score greater than or equal to this cut-off score. Sone have 
argued that the cut-off score roust be located at the upper extreme since 
flawless performance Is desirable. 



Nothing about criterion-referenced testing Implies any of this. 

That criterion-referenced testing does not depend on a cut-off score has 
been mentioned previously. Further, criterion-referenced testing does 
not Imply a value Judgment about whether flawless performance in desirable. 
It only seeks to assess what the behavior is. 

Hhcicher using cut-off scores with tests ia good or bad, is an 
empirical question although It is embedded In the ethical and decision 
network within which one operates. For example, given that certain 
terminal outcomes are desired and that an instructional sequence is 
specified, the question is, what level of performance is required at each 
point in the learning sequence in order to maxirite success at the rext 
point in the sequence and io on until the terminal learning is attained. 

. 

O This appears to be e transfer of learning problem and not »>n* which is left 




12 



11 



entirely to subjective Judgment* It its clear that such decisions cannot 
be based on poor information, such as a poorly constructed test, but must 
be based the empirical findings of instructional psychology. 

Related to criterion-referenced testing and mastery learning is 
the question of whether everyone needs to learn the same thing to the 
same degree and who imposses standards of competency. A reasonable dis- 
cussion of this issue and its ethical implications is beyond the scope 
of the presentation. (For a cogent discussion of this issue in another 
context see Bandura (1969) • Much of that discussion seems to apply to 
instruction.) Nothing in the nature of criterion-referenced testing 
implies that anyone necessarily meet a given standard of competency, only 
that such levels of competency be defined in terms of performance. 

A humanistic point of view would take into account the 
goals of the individual as related to the goals of society and allow the 
individual to participate in choosing and planning his learning experiences. 
If the individual desires to become a "master" and is motivated to achieve 
mastery, then of necessity we must provide him with the experiences which 
will facilitate his becoming a master and provide him with assessments 
so that he can evaluate his progress toward the goal he has chosen. To 
be sure, this point has been made by others. An interesting recent example 
of the successful application of behavioral analysis is that given by 
Zoellner (1969) with respect to the teaching of English composition. He 
states the problem in this way: 

"... the central failure of current compositional pedagogy . . . 
is its apparent inability to furnish the student-writer with 
anything but the most generalized specification for getting 
from one side of the writing situation (poor writing) to the 
other (good writing) . What is urgently needed is a peda- 
gogical technique which will supply the student-writer with 
a set of compositional specifications which are a) successively 
intermediate rather than ultimate, b) visible rather than 
invisible, c) uniquely adapted to the student's unique 

oblem, and d) behavioral rather than historical, i O 

to writing rather than the written word (page 
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The Need for Norm-Referenced Information 

So far this discussion has emphasized criterion-referenced infor- 
mation, The need for norm-referenced information as well as criterion- 
referenced information should be apparent. It is useful under cerf.sin 
circumstances to know not only what level of competency an individual or 
group has or does not have, but also how that competency is related to 
other individuals or groups which are similar In composition, have 
similar educational experiences, or which have similar aspirations* It 
is also important to kr.ow relative standing in groups that are basically 
different* 

But "useful 11 can only be interpreted in terms of purpose* In order 
to determine vh&t kind of information to collect or to emphasize* one needs 
to know what kind of decision needs to be made* In some decision contexts 
nora- referenced information is inescapable* It has been pointed out that 
in some parts of the world it may be that It is financially impossible to 
offer advanced education to all individuals* Here relative competency 
and relative standing with respect to all such applicants for education 
becomes one of the most important types of information that is needed for 
decision-making* Whether such a stance is valid is beyond the scope of 
this presentation* The answer to such a question, however* will determine 
to a large extent the type of information the educational decision -maker 
will need and the kinds of observations and data that will have to be 
collected. 




Criterion-Referenced Testing vs, Norm-Referenced Testing 

la criterion-referenced information better than norm-referenced 
information? One cannot discuss the usefulness of one measurement procedure 
over another without knowing the context within which that information is 
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needed and how it will be used* As Green (1969) has noted, considerations 
of measurement per se are wasteful in the overall decision-making process. 
Failing to consider the interrelationship between measurement and decision- 
making neglects the importance of deciding what additional data need to.be 
collected before adequate decisions can be made. 

There is a difference between taking measurement for scientific 
purposes and testing in instructional situations. The scientist is con- 
cerned with the identification and measurement of stable properties and 
variables. He seeks to determine general laws and rules for determining 
the relationships between these variables. He is discipline oriented 
and this dictates to a large extent the variables he chooses to measure 
and the way in which he measures them. In the practice of instruction 
one is concerned primarily about what each pupil desires to learn and 
how to maximize the learning he desires. What is learned is of primary 
importance and is usually defined in terms of acquired behavior and 
competence. Instruction provides the conditions by which this learning 
takes place. In a somewhat different context Lord (1968) speaks to this 
point • 



It should be clear that there are important differences between 
testing for instructional purposes and testing for measurement 
purposes. The virtue of an instructional teat lies ultimately 
in its effectiveness in changing the examinee. At the end, 
we would like him to be able to answer every test item correctly. 

A measurement instrument, on the other hand, should not alter 
the trait being measured. Moreover, . . . , measurement is 
most effective when the examinee knows the answers to only 
about half the test items, (page 2) 

It is a platitudinous assertion that an educational system should 
provide for individual differences and should allow students at every 
level of ability to develop and excel. Several patterns of instructional 
procedures for adapting to individual differences as they appear in the 
school can be identified (Cronbach, 1967). One pattern occurs where 
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educational goals and instructional methods are relatively fixed and 
inflexible* Individual differences are taken into account by dropping 
students along the way. The underlying rationale involved is that every 
child should M go as far a9 his abilities warrant . w A second pattern of 
adaptation to individual differences is one in which the prospective future 
role of a student is determined, and depending upon this role, he is pro- 
vided with an appropriate curriculum. For example, vocationally oriented 
students get one kind of mathematics and academically oriented students 
get a different kind of mathematics. Generally in this type of adaptation 
Individual differences the educational system has optional educational 
objectives, but within each option the instructional procedures are relatively 
fixed. A third pattern of adaptation to individual differences is one in 
which Instructional procedures are varied to accommodate the differences 
in each student. Different students are taught differently and the sequence 
of what is learned is not common to all students. One way in which this 
pattern is Implemented is to provide a fixed mainstream instructional sequence 
ind to branch students to remedial work when needed. Upon completion of 
remedial work the student is returned to the mainstream instruction, Another 
way of implementing this pattern is to begin with an assessment of a pupil's 
learning habits and attitudes, achievements and skills, cognitive style, 
etc. This information is jsed to guide the student through a course of instruc- 
tion that is uniquely tailored to his goals. Thus, students would learn in 
different ways and attain different goals. 

Each of these different patterns of instruction will require 
different kinds of measurement that result from different types of infor- 
mation requirements and instructional decision-making requirements. It 
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is Impossible then to apeak of the strengths and weakness of criterion- 
referenced or norm-referenced testing in a vacuum. The merits of any 
testing program lies In the extent to which it provides useful Infor- 
mation to the decieion-maker be he instructional designer, pupil, teacher, 
administrator, or the public at large 

Not only must this information be useful, but it must be usable 
as well. That is, the testing program must be designed Into the instruc- 
tional process so that the information that is required Is easily obtained 
and available in e usable form at the time a decision needs to be made. 
Built into such an instructional system oust be a procedure for conatantly 
updating and redefining the adequacy of the decisions being made and the 
information upon which they are based. 

Vhen viewed in this way, the distinction between testing and 
instruction becomes leaa dlatlnct, ao that the learner can look toward 
testing for feedback concerning his accomplishments and for guidance toward 
his chosen goal*. 
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